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Introducing the team 


Cleber Paiva de Souza 


Director 


cleber@ssys.com.br 
+55 19 99841-4242 
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Gabriel Dieterich 
Cavalcante 


Tecnhical Lead 


gabriel@ssys.com.br 
+55 19 98222-8118 
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SUSECON digital? l Ca 
SUSE Platinum Partner for Services and Training 


Many success cases together with SUSE in Brazil 


Specialists in Linux, monitoring and Salt automation 


About SSYS Many SUSE certified professionals in the team 


A software development company 











Challenges 


e Short time for project (40 days including go live, data load etc) 


e 15 days to deliver all SAP systems, register, updates, install applications, apply security 
baselines, Azure infrastructure, install SAP Applications, cluster configuration and testing. 


e Systems deployed (6 SAP Applications clusters) 


e Standalone Dev and QA stacks 
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Approaches 


e Usage of ha-sap-terraform-deployments 
e Do not support NetApp Files on Azure (on March/2021) 


e Need public IP addresses to run command on virtual machines 


e Created terraform template from the ground to validate SAP deployment scenarios 
e Added support for NetApp Files 


e Cloud init to run post setup and configuration 


e Azure Storage for SAP Medias 
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Terraform 


laaC (Infrastructure As a Code) 


Create infrastructure based on high level 
description files. 


In this case we used Terraform to create all 
infrastructure needed by SAP workload. 
Relying on Terraform to create, resource 
group, subnets, virtual machines and load 
balancers at Azure 
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Avoid Destruction 


With accurate plans we can check for instance 
if a subnet already exists and skip the 
creation. 


Only objects managed by Terraform are 
destroyed. 


Custom Cloud Init 


Terraform creates a custom cloud init on new 
machines. It is responsible for making all the 
adjustments inside the virtual machines and 
run the calls for salt-formulas to deploy SAP. 














SUSE Salt Formulas 


Masterless Mode 


Customer does not use Salt Master. We used cloud init to 
run SUSE Salt Formulas, with all pillars defined in /srv/pillar. 


https://github.com/SUSE/sapnwbootstrap-formula 
https://github.com/SUSE/saphanabootstrap-formula 


https://github.com/SUSE/habootstrap-formula 
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SALT 


PROJECT 
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terraform.tfvars (Part I) 


subscription id = "MY SUBSCRIPTION HERE" # Storage account # MISC 
client. 1d = "MY CLIENT ID HERE" Storage account = “Customer? CUSTOMER" sid adm password = "PasswOrdlz5" 
Cirent SecreL = "MY CHIENT SECRET HERE” storage tier = "Standard" sap adm password = "PasswÜrdl25" 
tenant-id = “MY TENANT ID HERE" Storage Xepl = "LRS" master password = "PasswÜrdl23" 
resoubce Group = "SUSELOnZI-Ccgs10735* sap media local — "/mnt/sapmedia" sapmnt path - "/sapmnt" 
location = UNGSLUS" sap media storage = "/hanamedia" srdadm user uid = "1003" 
admin username = "azureroot" sap media key — "YOUR CODE HERE" sidadm user gid = "1002" 
admin password = "PasswÜrdl23" sap media add fstab = "false" sapmnt inst media = "10.0.1.6:/var/sapmnt" 
swom folder = "/mnt/sapmedia/swpm /" 
# Network # Load balancer sapexe folder = "/mnt/sapmedia/kernel novo/partl/" 
vnet name = "vnet" create lb = "true" additional dvds - "/mnt/sapmedia/misc/" 
vnet addr = "10.0.0,.0716" create lb public ip = "true" 
subnet name = "subnet" lb private ip = "10.0.1200" # HANA 
subnet addr = "10.0.1.0/24" lb sku = "Basic™” sap hana host = "hana" 
sap hana 1p = "10.,0.1.100" 
# NSG rules # ASCS sap hana sid = "PRD" 
nog name = "nsg" sap ascs instance sid = "HAIT sap hana instance = "00" 
neg rules = { Sap ascs instance id = "017 sap hana password = "PasswÜO0rd123" 
name = "All" Sap eSCs FOoOUL User = *root” 
direction = "Inbound" sap.ascs root password = "PasswÜrdl25" 
access = "Allow" Sap ascs vip address = "I0.0.1,.200" 
protocol = "Tcp" Sap ascs vip hostname = "sáapsascs-vip" 
source port, range = "xm 
destination port range = "x" # ERS 
Source address prefix = "x" Sap ers instance sid = "HAL" 
destination address prefix e "s" Sap ers instance ad e "po" 
} esp Gre root user = "POOL" 
Gap ers roOL password = "Passwüsoclz3s" 
Sap ers vip address = "I0.0.1,.201" 
Sap ers vip hostname = "sap-ers-vip" 
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terraform.tfvars (Part II} 


# Load Balancing rules and probes # Azure Fencing Application 
Ip rules = 4 create fencing app = “true” 
probe name — "ssh,hawk" fencing app name = "Sap fencing app” 
probe porc = "22,7630" 
protocol = "Tcp, Tcp" # HA Cluster 
frontend port = "22,1630" cluster unicast — "true" 
backend port = "22,7630" cluster password = "PasswO0rd" 
enable floating ip = "false, false" 
idle timeout in minutes = "30,30" # SAP Monitoring 
load distribution = UDOUurCeLPPPrOLOCOloOUFCelPPIOLOGOL" enable monitoring = "true" 
j 
# NetApp Files 
# VMs SAP enable netapp files = "false" 
add pub ip — "true" netapp account name = "ocustomer-sap" 
add boot diag = “Crue” netapp pool name — "netapp-pool" 
machines = { netapp pool service level — "Premium" 
vm image = "SUSE:sles-sap-15-sp2-byos:gen2:latest,SUSE:sles-sap-15-sp2-byos:gen2:latest" netapp pool size = 4 
vm name = "sap-ascs-l,sap-ascs-z2" netapp volume name — "netapp-volume" 
vm size = "Standard Doll w2z,9—t5ndsrd Dell v2" netapp volume path — "sap-volume-path" 
vm ip = VO ed OL, LO... LOZ” netapp volume service level = "Premium" 
vm disk type = "Premium LRS, Premium LRO" 
wn net accel — "true,true" 
C loUOEHi t — "files/cloud-init-ascs.yaml,files/cloud-init-ascs.yaml" 
vm swap size = "4096,4096" 
wm reg code = "YOUR SUSE KEY,YOUR SUSE KEY" 
vm reg email = "oontacLecusctomer,.com,br,contacuscustomer.com.br" 
vm mount media - "true,true" 
j 
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Features added in Terraform 


— Availability sets 

— Proximity placement group 

— Cloud init template file 

— AzureAD Enterprise Application (ServiceCredential for fencing with fence azure arm) 
— AzureAD Role (for fencing) 

— Network security group (security rules) 

— CIFS mount for Storage Account (for SAP Medias) 

— NetApp Files for SAP data 


— Support for Azure scheduled events 
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fence azure arm (/usr/share/fence/azure fence.py) 


— Changes in Azure SDK after 15.x in azure- 
mgmt-resource 


— builti-in fence azure arm does not work 


—  AttributeError: 
ServicePrincipalCredentials' object has 
no attribute 'get token' 


— Needs installation of python3-azure-identity 


— Apply patch to 
/usr/share/fence/azure fence.py 
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2021902260 19:221:2951.000000000 +0000 
Z02lepaen, 232:17299.9542122494. TUDOU 


=== aure rencespy 
Tac SZUIe ence, py «new 
Ug =292,19 «4292,19 CC 
from msrestazure.azure active directory import MSlIAuthentication 
credentials = MSIAuthentication() 
elif cloud environment: 
i from azure.common.credentials import ServicePrincipalCredentials 


i credentials = ServicePrincipalCredentials( 
+ from azure.identity import ClientSecretCredential 
+ credentials = ClientSecretCredential( 


Glint 1d = COniigAPpPlicationId, 


= secret = config.ApplicationKey, 

= tenant = config.Tenantid, 

+ clrent secret = conil ig.-Applicationkey, 
F tenant id = confilgsTenantid, 


cloud envrronment-cloud environment 
) 
else: 
= from azure.common.credentials import ServicePrincipalCredentials 
= credentials = ServicePrincipalCredentials ( 
+ from azure.identity import ClientSecretCredential 
+ credentials = ClientSecretCredential ( 
Client 1d. = contig +<eppl cation ld, 
= secret = config.ApplicationKey, 
= tenant = config.Tenantid 
+ Client secret = OOnLticg.ApplicationBey, 
+ tenant id = conrtig,.Ténantid 


return credentials 








fence azure arm (/usr/sbin/fence azure arm) 


— Changes in Azure SDK after 15.x in azure- 


mgmt-resource 


— builti-in fence azure arm does not work 


— poweroff method not supported 


— Apply patch to /usr/sbin/fence azure arm 
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--- fusr/sbin/fence azure arm 2021902930 03250229. 7290020 175 +0000 
+++ /usr/sbin/fence azure arm.new 2024204938 :5590255,02109595 0000 
Gg 115,7 +115;7 gg 


Ir (optarons["--action"]|--"ofr"y3s 
logging.info("Poweroff " + vmName + " in resource group " + rgName) 
compute client.virtual machines.power off(rgName, vmName, skip shutdown-True) 
compute client.virtual machines.begin power off(rgName, vmName, skip shutdown-True) 
elit (optaons["--actaion"]--"on"): 
logging.info("Starting " + vmName + " in resource group " + rgName) 
compute client.virtual machines.start(rgName, vmName) 











Automation steps 


e Terraform launches infrastructure 


e Instances, public and private IPs, load balancer, storage, netapp files, azuread application, 
security groups, security rules etc 


e Cloud init runs: 
e Enable swap disk 
e Register SUSE 
e Update system 
e Mount SAP Media 
e Patch azure fence.py and fence azure arm 
e Configure salt-minion 
e Populate pillars 
e Apply Salt states (habootstrap-formula and sapnwbootstrap-formula) 


e Populate pacemaker CIB with SAP Netweaver primitives) 
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Some numbers 


e Full setup of SAP ASCS and ERS Replication servers 
e Infrastructure: 5 minutes 
e Appling patches, customizations and High Availability Bootstrap: 12 minutes 
* SAP Installation for ASCS and ERS: 14 minutes 


e Total: 31 minutes 


e Number of manual commands and interactions: O 
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DEMO » 


On this slide we'll jump into code 
and some demonstrations. 
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Thanks 


e Ronaldo Alves da Costa and Bruno Leonardo de Souza from Piracanjuba for supporting this 
work and trusting our services. 


e Carlos Motta from Microsoft for providing Azure labs for testing. 


e All code is available at https://github.com/s-svs/terraform-sap-cluster-automation 
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References 


e httos://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/high-availability- 





guide-suse-netapp-files 


e  https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/sap-high-availabilitv- 





guide-start 


e httos://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/high-availability- 





guide-suse-pacemaker 
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Thanks for watching 


Please contact us: 


cleber@ssys.com.br 
gsabriel@ssys.com.br 
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Continue the conversation in the SUSE & Rancher Community 
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Visit www.susecon.com for new technical content and information about 
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